• The population is the truth
• The sample comes from the population, but is generally smaller in size
• This means that not all cases from the population can be in our sample
• If not all information from the population is in the sample, then our sample may be wrong

• There are many flavours of sampling
• If we give every unit in the population the same probability to be sampled, we do random sampling
• The convenience with random sampling is that the missingness problem can be ignored
• The missingness problem would in this case be: not every unit in the population has been observed in the sample

• The problem is a bit larger

• We have three entities at play, here:

  1. The truth we’re interested in
  2. The proxy that we have (e.g. sample)
  3. The model that we’re running

• The more features we use, the more we capture about the outcome for the cases in the data

• The more cases we have, the more we approach the true information
  
  
  All these things are related to uncertainty. Our model can still yield biased results when fitted to \(\infty\) features. Our inference can still be wrong when obtained on \(\infty\) cases.

• The problem is a bit larger

• We have three entities at play, here:

  1. The truth we’re interested in
  2. The proxy that we have (e.g. sample)
  3. The model that we’re running

• The more features we use, the more we capture about the outcome for the cases in the data

• The more cases we have, the more we approach the true information
  
  
  

Core assumption: all observations are bonafide

When we do not have all information …

1. We need to accept that we are probably wrong
2. We just have to quantify how wrong we are

In some cases we estimate that we are only a bit wrong. In other cases we estimate that we could be very wrong. This is the purpose of testing.

The uncertainty measures about our estimates can be used to create intervals

An intuitive approach to evaluating an answer is confidence. In statistics, we often use confidence intervals. Discussing confidence can be hugely informative!

If we sample 100 samples from a population, then a 95% CI will cover the true population value at least 95 out of 100 times.

• If the coverage <95: bad estimation process with risk of errors and invalid inference
• If the coverage >95: inefficient estimation process, but correct conclusions and valid inference. Lower statistical power.

We can replicate our sample.

• A replication would be a new sample from the same population or true data generating model obtained by the same data generating process.
• If we would sample 100 times, we would get 100 different samples
• If we would estimate 100 times, we would get 100 different estimates with 100 different confidence intervals (e.g. 95% CI)
• Out of these 100 different intervals, we would expect a nominal coverage. For a 95% CI we’d expect 95 of them to cover the true population value.

Full sampling validation of a model’s inferences is a lot of work.

• it is the most robust way of obtaining inferential validity
• it is not always necessary

Under some general assumptions, we can use the same data to validate our model’s inferences and predictions.

• these assumptions can be met in practice
• but as soon as assumptions are made, we open the door to errors when these assumptions do not hold

Individual intervals can also be hugely informative!

Individual intervals are generally wider than confidence intervals

• This is because it covers inherent uncertainty in the data point on top of sampling uncertainty

In the decision to proceed with the launch, there was a presence of dark data. And no-one noticed!

Dark data

Information that is not available but necessary to arrive at the correct answer.

This missing information has the potential to mislead people. The notion that we can be misled is essential because it also implies that artificial intelligence can be misled!

We now have a new problem:

• we do not have the whole truth; but merely a sample of the truth
• we do not even have the whole sample, but merely a sample of the sample of the truth.

There are two sources of uncertainty that we need to cover when analyzing incomplete data:

1. Uncertainty about the data values we don’t have:
   when we don’t know what the true observed value should be, we must create a distribution of values with proper variance (uncertainty).
2. Uncertainty about the process that generated the values we do have:
   nothing can guarantee that our sample is the one true sample. So it is reasonable to assume that the parameters obtained on our sample are biased.

A straightforward and intuitive solution for analyzing incomplete data in such scenarios is multiple imputation (Rubin, 1987).